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Performance Report Requirement, Part B and the Continuous Improvement 
Monitoring and Targeted Monitoring" (Ruth Ryder of the Office of Special 
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Background and Purpose 



A defining element of the current educational reform movement is its focus on the setting of 
standards with performance goals and indicators developed for the purpose of reporting on 
and tracking the progress of students. When the Individuals with Disabilities Education Act 
(IDEA) was amended in 1997, a new section required that every state: 

(a) Has established goals for the performance of children with disabilities in the 

State that - 

(1) Will promote the purposes of this part, as stated in §300.1 ; and 

(2) Are consistent, to the maximum extent appropriate, with other goals and 
standards for all children established by the State; 

(b) Has established performance indicators that the State will use to assess progress 

■toward achieving those goals that, at a minimum, address the performance of 
children with disabilities on assessments, drop-out rates, and graduation 
rates; 

(c) Every two years, will report to the Secretary and the public on the progress of the 

State, and of children with disabilities in the State, toward meeting the goals 
established under paragraph (a) of this section [34 CFR §300.137], 

The first Biennial Report required by this section was filed by all states in December 1999. 
Project FORUM completed an analysis of those reports 1 , and found a considerable amount of 
variability among states not only in the content, but also in the meaning and use of the terms 
‘goals’ and ‘indicators.’ 



In October 2000, the U. S. Department of Education Office of Special Education Programs 
(OSEP) funded a new project, the Educational Policy Reform Research Institute or EPRRI, 
to research and analyze the impacts of accountability-based reforms on students with 
disabilities and special education. Institute staff proposed an activity to gather state and local 
input on performance goals and indicators and, as a result of coordinated planning with 
OSEP, it was decided that Project FORUM would work with the Institute to convene a 
meeting to examine the topic. 



The policy forum, Performance Goals and Indicators for Special Education, was held at the 
Hilton Crystal City in Arlington VA on May 14-16, 2001 as part of the cooperative 
agreement between Project FORUM at the National Association of State Directors of Special 
Education (NASDSE) and the U.S. Department of Education’s OSEP. The policy forum was 
planned and conducted in conjunction with EPRRI to discuss the feasibility and desirability 
of developing a cohesive data-driven system of performance goals and indicators that can be 



1 Aheam, E. M. (2001, January). Performance goals and indicators. Alexandria, VA: National Association of State 
Directors of Special Education. 
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used for both special education program improvement at local and state levels, and for 
federal data requirements. The intended outcomes of the meeting were a unified indicator 
system, and a set of potential indicators with consideration for their collection and use. 



Logistics of the Meeting 

Planning and Selection of Participants 

Project FORUM worked with OSEP and EPRRI to identify participants for the 
meeting. The invitees included the State Directors of Special Education and one local 
director of special education from each of the four EPRRI focus states, representatives of the 
National Center on Educational Outcomes (NCEO) and the Educational Development 
Center, Inc. (EDC) who are partners in EPRRI, other senior advisors to the project, and 
representatives of other stakeholders including NASDSE, parents, the National Association 
of Protection and Advocacy Systems (NAPAS), universities, Westat, the Council of Chief 
State School Officers (CCSSO), and the U. S. Department of Education. The final list of 
participants is attached to this report as Appendix A. 

Process of the Meeting 

Materials provided to participants in advance as background for the forum included a 
copy of the FORUM QTA, a copy of the draft EPRRI topical review entitled Performance 
Goals and Indicators in Special Education , description of the model of accountability 
developed by NASDSE 2 , a copy of the NCEO Framework for Educational Accountability 3 , 
and a copy of the cluster areas currently used in OSEP monitoring 4 . 

The plan for the meeting included presentations by U. S. Department of Education and 
EPRRI staff to provide an orientation about federal data requirements, and an overview of 
the topic of performance goals and indicators. Participants worked in two small group 
sessions to discuss domains, types of indicators, and data needs from state and local 
perspectives. The discussions were analyzed and, at the concluding session, a framework for 
considering indicators was presented and discussed. A copy of the agenda is included in this 
report as Appendix B. 

The remainder of this report provides a summary of each presentation, a synopsis of the 
small group discussions, and a description of the outcomes of the meeting. 

Summary of Presentations 

I. Biennial Performance Report Requirement, Part B and the Continuous Improvement 
Monitoring and Targeted Monitoring. Presentation by Ruth Ryder, Division Director, 



2 NASDSE 's Model of Accountability is available as Publication #PFR555 through the NASDSE website at 
www.nasdse.org 

3 NCEO documents are available at the website: www.coled.umn.edu/NCEO/OnlinePublications 

4 For a draft copy of the cluster areas, see the website: www.dssc.org/frc/nmDp/whatsnew.htm 
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Monitoring and State Improvement Planning Division, OSEP, U. S. Department of 
Education. 

Continuous Improvement Monitoring (CIM) 

Development of this approach has proceeded over about the past three years. A new 
stage is now beginning and a new graphic will reflect the direction. The new graphic looks 
like a target with “improved results for children with disabilities and their families” as the 
center. The center is then circled with state-level activities including the self-assessment and 
improvement planning. The next wider circle includes the steering committee and technical 
assistance providers, and then OSEP occupies the outside circle. The process is depicted in 
the following graph: 

CONTINUOUS IMPROVEMENT MONITORING PROCESS 
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There are five cluster areas each for Part B and Part C. The Part B areas are: 

> Parent involvement; 

> FAPE (free appropriate public education) in the LRE (least restrictive 
environment); 

> Secondary transition; 

> Early childhood transition; and 

> General supervision. 

The Part C areas are: 

> Family centered services; 

> Child find and public awareness; 

> EIS (early intervention services) in the NE (natural environment); 

> Early childhood transition; and, 

> General supervision 

For each cluster area, there is an overall objective, a component, indicators, and 
recommended data. At this time, the cluster areas are being reviewed. Some of the items 
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appear to have been misplaced and some were too extensive with data not critically related to 
the component. The new charts will have the cluster, the component, the indicators, the data, 
and legal authority in a more condensed form. 

Focused Monitoring 

A stakeholder group (parents, advocates, State Directors, Part C Coordinators) was 
convened by OSEP starting in 1 998 and the development of the CIM approach resulted from 
their input. Then, in November 2000, the group was brought back together to review the 
changes. This group’s deliberations resulted in a proposal for Focused Monitoring that 
would form the basis for further changes in the process. The key principles would include: 

**** Priority areas - narrowed down to critical ones; 

♦ Indicators - measuring performance in each priority area; 

*t* Benchmarks - matching state data to its own prior performance; and, 

*1* Triggers - levels for determining whether a state needs further work in one area 
or has developed a successful practice to share with others. 

At the next meeting in July, benchmarks and triggers will be developed. Stakeholders are 
now circulating this proposal to obtain input from all constituencies. 

Biennial Performance Report : 

This report combined the existing requirement of a performance report with the new 
Part B requirement that states report on the progress of students with disabilities in meeting 
performance goals and indicators set by the state. The first time it was filed was in the fall 
2000. States were instructed to report on the following: 

♦ Performance Goals and Indicators, including dropout rates, graduation rates, and 

participation and performance on assessments; 

♦ Long range suspensions and expulsions; and, 

♦ Disproportionality 

The information that was gathered in this first filing was not in a form that could be 
used by OSEP. The reports ranged from 5 to 574 pages in length and varied greatly in 
content. OSEP is now meeting with groups about how the Biennial Performance Report can 
be improved. 

II. Annual Report to Congress; Government Performance and Results Act (GPRA); Office of 
Civil Rights (OCR) Data. Presentation by Lou Danielson, Division Director, Research to 
Practice Division, OSEP, U. S. Department of Education. 

Annual State Data Requirements : 

Every state is required under IDEA to provide data in a number of areas. Under Part 
B, data collections include: 

^ Child count - the number of students with disabilities receiving special 
education and related services under IDEA; 

•C Educational environments - the extent to which students with disabilities are 
being served with their non-disabled peers; 

^ Personnel - counts of personnel employed and contracted in full time 
equivalents to serve students with disabilities; 

'A Exiting - the number of students in each state ages 1 4-21 who exit special 
education during a school year; and. 




Performance Goals and Indicators for Special Education 
Project FORUM at NASDSE 



9 



Page 4 
August 6, 2001 



^ Discipline - the number of students in each state who are removed to an 
interim alternative education setting, the act precipitating those 
removals, and the number of children who are subject to long-term 
suspensions or expulsions. 

Under Part C, data requirement areas include: 

^ Child count - the number of infants and toddlers receiving early intervention 
services under IDEA and the number who are at-risk of developmental delay; 

S Services - a count of the number of early intervention services provided to 
infants, toddlers and their families; 

^ Exiting - count of infants and toddlers who stopped receiving early 

intervention services because of program completion or for other reasons; 

^ Settings - count of infants and toddlers and their families receiving early 
intervention services by primary setting of the service; and, 

^ Personnel - the number of personnel employed and contracted in full time 
..equivalents to provide early intervention services. 

Government Performance and Review Act ( GPRA) 

The GPRA was passed in 1993 to improve the efficiency and effectiveness of federal 
programs. All departments must report annually on goals and indicators. Performance on 
these indicators is used in determining budget allocations. Goals, objectives and indicators 
have been set by OSEP for Parts B, C, and D of IDEA. Participants were provided with a 
summary chart of these three sections of the Department of Education GPRA Plan that is 
available in a variety of formats on the OSEP website at the following address: 
http://www.ed.gOv/pubs/AnnualPlan2001/index.html#special . 

Office of Civil Rights (OCR) Data Collection 

This data collection is over 30 years old and is often known by the name of the 
forms involved (101 and 102 or the District Summary Report and Individual School Report). 
It almost always involves only a sample of schools with the data extrapolated to the entire 
population. However, in the year 2000, this data collection included all districts and can be 
analyzed down to each school building in every state. The District Summary Report has 
general information on districts and schools, includes child count for both IDEA and 504, 
and contains information about state and district tests that students are required to pass 
including participation and pass/fail data. The Individual School Report includes information 
on specific groups: gifted, LEP (Limited English Proficient) students, teachers, child count 
by disability and race and gender, special education, discipline of students with and without 
disabilities, placements, and mental retardation using subgroup definitions. 

III. Overview of Setting Performance Goals and Indicators. Presentation by Margaret Goertz, 
Consortium for Policy Research in Education and EPRRI Senior Policy Advisor 
Definition of Indicators and Indicator Systems 

An educational indicator is “an individual or composite statistic that relates to a basic 
construct in education and is useful in a policy context” (R. Shavelson, L. McDonnell, and J. 
Oakes, (Eds.), Indicators for Monitoring Mathematics and Science Education, RAND, 

1989). Educational indicator systems are designed to provide information both on the distinct 
components of the educational system and on how the relevant components work together to 
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affect educational outcomes. Choice of the system and the indicators should be based on the 
intended use of the system - otherwise someone could drown in data. 

In designing an education indicator system, key dimensions are identified first and they must 
be based on a conceptual model of the system. One example is the Inputs, Processes, Outputs 
system described by Andrew Porter in “Creating a System of School Process Indicators, 
Educational Evaluation and Policy Analysis, 1991.” Then, a manageable set of indicators is 
selected and many decisions must be made such as the level for reporting data, the frequency 
of data collection, etc. 

Policy Uses of Education Indicator Systems: 

> To report condition or status of different components of the educational 

system (inputs, process or outcomes) at different levels. This may be 
administrative or public reporting; 

> To compare the condition of the system with other schools, districts, states, 
.countries; 

> To monitor the system’s progress in attaining certain specified goals or 

policies; 

> To evaluate the impact of policies and programs on the educational system 

(program improvement); 

> To identify potential problems in the educational system; 

y To hold teachers, schools, and/or school districts accountable for inputs, 
processes, and/or student outcomes; and, 

> To reward good performance or seek to remedy poor performance. 

Design and Implementation Issues 

There are many issues that are involved in such a system. Any system will be used for 
multiple purposes, but, indicators should be used for only the purpose for which they are 
designed. Some of the most critical issues involved in the design and implementation of an 
indicator system are: 

❖ Feasibility of data collection and reporting; 

❖ Validity (Are the right things being measured - Does the instrument capture 

the right aspects of what is measured?) and reliability 

♦♦♦ Transparency — of particular note if different measures are combined into an 
index. It is important to make this understandable by decomposing its 
components. 

*1* Comparability — as states develop their own systems and goals and measures, 
there is a challenge to comparable data on several levels. Every state uses a 
different assessment systems; different inclusion policies. States have 
flexibility in determining performance goals for students - the dilemma is 
what is good enough to be accepted as proficient? 

♦> Parsimony - there are more questions than a system can manage to answer and 
choices must be made. 

❖ Data burden - on schools, districts and states to collect, on others to house and 

analyze them, and those who report. 

❖ Organizational responsibilities - who? what level? 

What does research tell us? 
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The relationship of inputs such as teacher quality to student outcomes is unclear: no 
relationship was found in many cases. There are questions about the unit of analysis, an area 
of a lot of controversy. 

We have some outcome indicators, e.g., students who meet benchmarks, but how is 
this to be measured - only by tests? What about other factors? There is no universally 
accepted small set of indicators that will lead to desired outcomes - we have not yet 
identified them. We do not have the research that will tell us what we want to know on this 
topic. In Tennessee, identifying teacher quality through student test results is a very 
controversial example of one effort. 

We also do not yet know how graduation rates relate to teacher quality. It appears that 
there is a need for an interim measure such as course taking. Relationships have been shown 
between types of courses taken and success in college admission. There is also a body of 
research that shows the importance of school completion on lifetime earnings and the 
association of dropping out with retention in grade and course failure. These are important 
factors to track. We are beginning to understand the conditions under which dropouts occur. 

Types of indicators are also important to understand. For example, see the report by 
Mary Moore (. Monitoring School Quality)fov a discussion of outcome indicators vs. process 
indicators and others in areas of teacher quality. However, questions of feasibility remain: 
how much information do we need to capture “quality practice”? Such questions often lead 
to things like a 550 page report. 

IV. Overview of Title I Accountability Requirements. Zollie Stevenson, Jr., Education 
Program Specialist, Office of Elementary and Secondary Education (OESE), U. S. 
Department of Education. 

The Title I accountability framework is designed to hold schools and districts 
accountable for improving student performance. It consists of state content standards, student 
performance standards, and aligned assessments to measure progress toward meeting state 
standards. Students with disabilities and LEP students are included and there are systems for 
rewarding successful schools and districts, and for identifying and intervening in schools and 
districts that fail to meet “adequate yearly progress” (AYP) growth targets. 

OESE believes that a strong assessment program is the best tool to make sure that no child is 
left behind. States must develop assessment systems that are aligned to their state standards 
and administer these assessments by spring 2001. Some states assess in more areas than 
mathematics and reading, but federal law requires only those two at this time. Assessments 
must meet technical standards for quality. Title I requires that a State’s final assessment 
system be the primary factor in determining AYP, although States can use non-test items 
such as school completion rates in their definition of AYP. The state determines its own 
targets, but once they are determined, the state must decide which have made the cut and 
what steps will be taken for those that did not meet the target. Schools not meeting AYP are 
targeted for school improvement, and schools that fail to improve can be subject to corrective 
action. As of now, 12 states have full approval for their assessment systems and 40 do not 
(total includes Puerto Rico and the District of Columbia). 
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States must report results for schools disaggregated by each major racial/ethnic group, 
gender, English proficiency status, migrant status, students with disabilities compared to non- 
disabled students, and by economically disadvantaged students compared to students who are 
not economically disadvantaged. School profiles must be issued and shared with the public 
that report disaggregated results of student performance. 

If the state measures the performance of all children, the same assessments must be used to 
measure the performance of Title I students. Reasonable adaptations and accommodations 
for students with diverse learning needs are required to be provided where they are necessary 
to measure the achievement of those students on state standards. English language learners 
must be assessed, to the extent practicable, in the language and form most likely to yield 
accurate and reliable information on what they know and can do to determine their mastery 
of skills in subjects other than English. Title I requirements pertaining to students with 
disabilities go beyond those in IDEA in that Title I requires that the scores of students with 
disabilities be included with the scores for all other students in making school and district 
accountability decisions. 

Small Group Sessions 

The purpose of the small groups sessions was to discuss issues as they pertain to different 
levels. Margaret McLaughlin introduced and discussed a graphic representation of the 
current status of data requirements. 




Data is needed at all three levels. However, what we now have is a heavy set of requirements 
at the federal level, less at the state and the least at the local level. But, the need for data to 
guide instruction is greatest at the local level where the availability is the lowest. It appears 
that the triangle needs to be turned upside down. 



Performance Goals and Indicators for Special Education 
Project FORUM at NASDSE 



Page 8 
August 6, 2001 



Participants were divided into four groups, two each representing the state level and the local 
district level. A sample set of indicators was provided as a starting point for the discussion. 
Groups were asked to focus on five areas. For each area, questions discussed were: 

1 . What is the purpose for each indicator? 

2. Which indicators meet the purpose? 

3. How would the data be used? 

4. Which is the most important indicator? 

The discussions of the two groups at each level were synthesized into one summary report 
for the level. 

State Level Small Group Summary: 

The state groups started with outcomes, looking at key outcomes and system 
variables. It was noted that each state has different requirements and thus each one’s purpose 
will differ. In general, the purpose is to look at the education system for improving results of 
students with disabilities. Some special education key indicators that should be included for 
the state-level are: 

• Assessment performance and rates of participation 

• Percent earning high school diploma 

• Percent dropping out of school 

• Participation in post-secondary education or employment 

• Classification and declassification rates 

• Measurement of LRE 

• Proportions by ethnicity/race 

• Complaints 

Local Level Small Group Summary: 

The local groups agreed that local level data must be rich and reflect the diversity of the 
students and their educational environments. It is critical that people in the trenches 
understand the relevance of the data they are asked to collect. School site data needs are the 
greatest because this is the level most sensitive to program improvement. Student 
performance data should drive the indicator system. A set of unifying objectives and 
indicators should be set by the Board of Education and applied across all schools in the 
district. They should be used for accountability as well as program improvement. Although 
some models exist, in many places, districts are a muddle of the process. In terms of 
indicator domains: 

• Context indicators are critical 

• Student performance data are the most important 

• There is little or no confidence in the traditional indicators such as teacher credentials, 
but personnel indicators should include turnover and mobility. Related service 
providers and paraprofessionals should be included. 

• Educational process indicators are very important and should include items such as 
improvement plans, staff evaluation, collaboration across departments, alignment of 
administrative and supervisory structures. 
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• Schools receive all data disaggregated by students with disabilities with a clear 
feedback loop from school to district to ensure accountability 

• Social/behavioral and parent participation is ranked higher than other domains. 

Overview of Small Group Work: 

Both groups want information that allows them to make better decisions - “actionable 
information.” The state level indicators should be few and very pointed and used as alerts for 
further exploration when a problem is indicated. The local level needs more specific 
indicators. Some indicators should be used for accountability purposes, others for incentives 
or other types of reporting. Sets of data can have powerful effects on raising the expectation 
level for student performance. The frequency of data collection should be determined by its 
use. 

Indicators should be warning bells or “flags.” Where you stand determines what you 
consider to be an appropriate indicator. There are differences of opinion on the use of 
indicators: some say nothing else matters if students are learning. Collection of much data at 
this point is done for compliance reasons and often it is not used for program improvement. 
There are problems around data collection and how the data are used. One of the biggest 
problems is comparability. 

Concluding Session 

At the concluding session, Margaret McLaughlin, EPRRI co-principal investigator, 
summarized the deliberations and presented A Framework for Considering Indicators : 

> Data are collected and used for different purposes at different levels. 

> Many different forms of data can be collected. 

> Data tend to flow from the local level toward the federal level. 

> Data elements have different meaning and value at different levels. 

> All data are potential information. 

> Data become information when they are used for a particular purpose in a given 
context. 

> Indicators are a specific type of information used for making comparisons or 
evaluations — over time, with a standard, among or within groups. 

> All data are not indicators. 

> Indicators may be comprised of only one or more than one data element. 

> Data may be used to develop different indicators in different contexts. 

> Some data may be used in multiple contexts. 

> Indicators must be considered on the basis of their. . . 

❖ Importance to various stakeholders 

❖ Validity for decision making 

❖ Reliability 

*1* Potential for use or action vis a vis external forces, demographics, or other 
contexts. 
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Some special considerations include: 

□ Standard definitions and more specificity for key data elements will be needed 

to support comparability. 

□ Resources, time, money, expertise, human capacity, technology, etc., become 

big issues when data elements are added. 

□ Feasibility and links to resource issues include who owns the data, and can the 

variable be measured. 

□ Timing and coordination of data collection is critical for local decision makers 

to make improvements in a timely fashion. 

□ Value placed on the data at different levels can impact validity. 

Participants commented that the effort needs to be put at making information systems 
effective in making differences for students. This would suggest that it is the district and state 
that are more critical than state and federal. Sometimes data are misleading and confounded. 
For example, a large number of students who receive only speech therapy can serve to make 
LRE data appear very positive because those students are all in the general education 
classroom while the district may really be poor in placement for other types of students with 
disabilities. In addition, different communities have different values and want to have 
different types of information about their schools. Often, students with disabilities are not 
considered when changes are proposed, but including them in data collections can bring 
significant change. Indicators should not be used for simple compliance decisions alone, but 
rather as a tool for identifying areas where there should be a closer look. Indicators must be 
viewed as a start of a conversation, not the end. 

Next Steps 

This proceedings document will be made available on the NASDSE and EPRRI websites: 
www.nasdse.org and www.eprri.org . 

EPRRI will revise and complete the Topical Review that was provided in draft form as a 
background paper for this forum. The Topical Review, entitled “Performance Goals and 
Indicators in Special Education,” will be posted in August 2001 on the EPRRI Website, with 
a limited number of print copies also available. 
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This document, along with many other FORUM publications, can be downloaded from the Project FORUM at NASDSE web address: 



http://www.nasdse.org/forum.htm 



To order a hard copy of this document or any other FORUM publications, please contact Carla Burgman at 

703-519-3600 ext. 312 or caria@nasdse.org 
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